EQUALITY OF SCHUR'S Q-FUNCTIONS AND THEIR SKEW ANALOGUES 



HADI SALMASIAN 



Abstract. We find a simple criterion for the equality Q\ = Q^/ v where Q\ and Q M /„ are Schur's 
Q-functions on infinitely many variables. 



1. Introduction 

Schur's Q-functions are very interesting analogues of the (standard) Schur functions s\ in sev- 
eral combinatorial and representation-theoretic contexts. Examples of their analogy include the 
shifted RSK correspondence, the shifted Littlewood-Richardson rule, and the character theory of 
representations of queer Lie super algebras. In this note we study when certain shifted Littlewood- 
Richardson coefficients (the in the language of \Stl\ ) are zero or one. Studying questions of 
the same nature has been of interest to a number of authors. In particular, one should mention 
Stembridge's recent classification of multiplicity-free products of Schur functions |St2j which was 
generalized to P-functions in |Besj . The related questions as to when two ribbon Schur functions 
are equal and when a Schur function is equal to a skew Schur function were answered in |BTWj 
and Wi . Here we show that the latter problem has a simple answer for Schur's Q-functions as well. 



This note is organized as follows. In the next section we give all the required definitions. In the 
third section we prove our main result. 

Acknowledgement. The author thanks the referee for reading the paper very carefully. 

2. Schur's Q-functions and shifted tableaux 



Our notation is compatible with Stembridge's paper |Stlj . A strictly decreasing sequence A = 
{Ai > ... > Afc} of positive integers is called a distinct partition of n if the sum of the A,'s is equal 
to n. The A,'s are called the parts of A. A partition is represented by a shifted Young diagram as 
follows: there are Aj boxes in the i-th. row, after i — 1 empty positions. We denote this diagram by 

Example 1. Let A = {7 > 4 > 2 > 1}. Then the diagram D\ which represents A is given below. 



Consider an ordered alphabet 

^ = {T<1<2<2<3<3<---}. 

The letters 1,2, ... will be referred to as marked, whereas the letters 1,2, ... will be referred to as 
unmarked. 

Definition 1. By a generalized shifted Young tableau (GSYT) of shape D\ we mean a filling of a 
given Young diagram D\ with letters from A such that the following properties hold: 
• The rows and columns are weakly increasing. 
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• Each row contains each marked letter at most once. 

• Each column contains each unmarked letter at most once. 

Let T be a given GSYT. We define x T to be the monomial x^x^ 2 ■ ■ • where a s is the total 
number of occurrences of s or s in T. Schur's Q-function Q\ is equal to 

T 



v x 

T 

where the sum is over all GSYT of shape D\. The sequence ai, 02, ... is called the content of T. 

Now suppose A and fi are two distinct partitions. Assume that for each i, A, > /Xj, so that 
lies inside D\. The shifted skew diagram associated to A/// is the set-theoretic difference of D\ and 
Dp. It is represented by Dxu- 

Example 2. Let A be as in Example ^ an d let \i = {4 > 2}. Then Dw^, is represented by the 
following skew Young diagram: 



We may assume that D\/^ is placed on the Cartesian plane in the usual way such that the centers 
of boxes lie on the lattice of points with integer coordinates. Let M Xty denote the box whose center 
is at the point with coordinates (x,y). The following easy lemma includes some basic properties 
of shifted skew diagrams. 

Lemma 2. Let Dx/^ be an arbitrary shifted skew diagram on the Cartesian plane. For any integer 

V defin€ Ry = {®x,v\®x, v e D x/fl }. 

• If B U) „ € Dx/n and 7^ then there exists an integer t > u such that Mt iV +i € Rv+l- 

• For a fixed v such that R v 7^ 0, let u be the smallest number for which M UyV £ R v . Assume 

€ F>x/ fl - Then for any v' < v + 1 such that R v > / 0, the following statement is 

true: 

o M u+v _ v /y 6 R v t and for any s if'M s y 6 R v > then s > u + v — v' . 

Schur's skew Q-function Qx/^ is equal to a summation similar to where the summation is 
now on all shifted skew tableaux T, with underlying diagram Dx/^ filled by the alphabet A, such 
that they satisfy the properties of Definition ^ The function Qx/^ can be expressed as a linear 
combination of the functions Q v for various v as follows. We have 

(2) Qx/» = Y.tivQ" 

where the summation is over all distinct partitions v. Here f* v is the number of amenable tableaux 
of shape Dx/^ and content v. We define the amenable tableaux in Definition |H1 below. However, 
before doing so, we need some notation. For a given (possibly skew) GSYT such as T, the row word 
of T is the word obtained by reading the rows of T consecutively from left to right starting with 
the bottom row. We denote the row word of T by w(T). Now, let w = w\ ■ ■ ■ w n be an arbitrary 
word of length n such that for any s we have w s 6 A. Define a function mj(j) as follows. 



number of times i appears if 1 < j < n 

among w n -j+i, ...,w n 



iTii(n') + number of times i appears if n + 1 < j < In 
among wi, Wj- n 
By convention, we assume mj(0) = for any i > 0. 

Definition 3. Let k > 1 be an integer. A word w = w± ■ ■ ■ w n is called k-amenable iff it satisfies 
the following properties: 
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• For any j £ {0, ...,n - 1} if m k (j) = m k -i(j) then w n -j £ {k, k}^ 

• For any j G {n,...,2n- 1}, if m k (j) = m k -i(j)_ then Wj- n +i £{k,k- 1}. 

• If j is the smallest number such that Wj € {k, k}, then Wj = k. 

• If j is the smallest number such that Wj & {k — l,k — 1}, then Wj = k — 1. 
A word w is called amenable if it is k-amenable for any k > 1. 

Remark. Suppose w = w\ ■ ■ ■ w n is /c-amenable for some k > 1. Then it follows from Definition 
El that if m k -i(2n) > then m k (2n) < m k _i(2n). (To prove this, first we show that Definition E] 
implies m k -i(j) > rn k (J) for any j E {1, 2n}. Then we pick the largest j such that Wj £ {k, k}, 
and we show that Definition El implies that there must exist a ji > j such that Wj 1 = k — 1 . The 
details of the argument are left to the reader.) Consequently, if w = wi • • • w n is amenable, then 

m\{2n) > rri2{2n) > m^{2n) > • • ■ 
Definition 4. A given GSYT is called amenable iff its row word is amenable. 

3. The main result 

In this section we prove the main statement of this note. 

Definition 5. A shifted skew diagram Dx/^ is called strange iff Qx/fi = Qu for some distinct 
partition v. 

Theorem 6. D^/^ is a strange diagram if and only if X/fj, = X/~p where 

• A is arbitrary and /! = {}. 

• A = {m > m — 1 > • • • > 1} and ~fi = > • • • > /i;} where < I < m — 1. 

• X = {p + q + r>p + q + r — 1 > p + q + r — 2 > ••• > p} and 

~fi = {q > q — 1 > • • • > 1} where p, q, r are integers such that p, q > 1, r > 0. 

• X = {p + q>p + q— 1 > ■ ■ ■ > p + q — r} and JI = {q > q — 1 > ■ • ■ > q — r} where p, q, r 
are integers such that p > and q > r > 0. 

Remark. The reader should note that there are overlaps among the cases for special values of 
p, q, r. Moreover, whether or not Dx/n is strange only depends on A/// and not on A and fj, individ- 
ually. Theorem identifies D by identifying all possible differences X/fi, but not all possibilities 
of A and ^. The latter problem is not hard once we have Theorem 

The proof of Theorem El will be given throughout this section. From equation (J2J it follows that 
Dx/fj, is strange if and only if there exists a unique amenable filling of D\/^- Our approach is to 
rule out various possibilities for the shape of D by demonstrating the existence of at least two 
different amenable fillings in each case. 

3.1. An algorithm for finding an amenable filling. We give a simple algorithm to construct 
an amenable tableau of any given shape D\/^- The output of the algorithm is an amenable tableau 
of content v for some distinct partition v. Note that v is generated by the algorithm and is not an 
input. 

Notation. Let be a box in a diagram. The operation of filling (or replacing the current entry 
of) M XjV by the letter w € A is represented by <— w. The operation of filling (or replacing the 
entry of) M XtV by the current entry of B^/y is represented by M XtV <— B,j./y . 

The algorithm is given below. 
Step 1 . Set k = 1 and D^j = D x/fJi . 

Step 2. Set P k = {M Xiy |B x , y G D { $ but B^-i^+i £ D%) }. 



4 



HADI SALMASIAN 



Step 3 . Put a k or a k in any of the boxes in P^ according to the following rule: 

k if B X) j,_i £ P fc 
fc if 1^,^-1 g P fc 



,(fc) 



Step 4. Remove all boxes of -Pfc from D^,. 

boxes. If DwJ 1 ' has no boxes, then s' 
Step 5 . Increase A; by one. Go back to Step 2 



Let D^j^ be the diagram obtained after removing 



Let A: be a positive integer. Then each connected component of Pk forms a "path" of boxes 
in a connected component of the diagram D . (A connected component of (or P^) is a 

maximal subset of boxes of D^) (or which can be ordered in a sequence so that each box has 
a common edge with at least one of the boxes preceding it in the sequence.) Each of these paths 
can be directed as follows. Let Q be a connected component of Pk- We know that Q is a path of 
boxes. The first box of this path is the box M x v G Q such that M x ^ y+ i ^ Q and M x+ \^ y ^ Q. The 
last box of Q is the box M X y G Q such that B^-x^ ^ Q and B X)2/ _i ^ Q. 

Example 3. The first and last boxes of the following path are marked with a dot and a cross 
respectively. It is traversed from the dotted box to the crossed one. 









• 






X 









Lemma 7. For any k > 1, P^ 



{M Xt y\n X: y G -Da/aj and 



Example 4. Let A = {7>5>3>2>1} and \i 
following amenable filling. 



G -Pfc-i}. 

{4 > 1}. Then the algorithm provides the 



Lemma 8. Let k > 1 and 

• Bj^+i G £> A/At and '. 

• IfB x> y +1 G P fc _x f/ien 

Proof. Left to the reader! 



G Pfc. T/ien 

K,y+1 G -Pfc U -Pfc- 





T 


1 


1 


1 


1 


1 


2 




1 


2 


2 






2 


3 








CO 





y+ i G Pfc-i a s well. 



□ 



The following lemma is a very simple but useful criterion of amenability. Its proof is left to the 
reader. 

Lemma 9. Let T be a GSYT of shape Dx/^ and let w = w(T) be the row word of T. For 
any integer k > 1, form the word from w by dropping all the letters in w which are not in 

22322112T11 then 



{k, k, k — 1, k — 1}. (For example, if w 
if and only if is either k-amenable or empty for any k > 1. 



223222. ) Then w is amenable 

Lemma 10. The filling of the boxes obtained by the previous algorithm is amenable. 

Proof. Properties of Definition ^ are satisfied trivially. It remains to show that this filling is 
amenable. By Lemma 03 it suffices to check fc-amenability of w(T)( k ' for any k. It is easily seen 
that if D\/n is disconnected but each connected component of T is amenable, then T is amenable 
too. Therefore we can assume L)\i^ is connected. Next we show that w(T)^ is amenable. The 
argument in the general case is similar. Let the length of w(T)^ be n. Amenability of w(T)^ 
follows from the following facts: 

• For any box filled by a 2, there is a box above it in the same column which is filled by a 1. 
This implies that mx(j) > m2(J) for any j < n. 
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• Since the last entry of the path P\ is filled by a 1 and there is no 2 below it, we have 
mi(n) > 7B2(n). 

• There is exactly one 1 in every row of D\/n m which Pi has boxes, except for the lowest 
row among them. The same statement holds for Pi and 2. 

• Set Xi = {y\ for some x,M X)V G Pi}. Then 

X\ = {s \s is an integer and c\ < s < 02} 
for integers c\ > c%. Moreover X2 C {s — 1 \s E Ai}. 

□ 

3.2. Disconnected diagrams. A slight modification of the algorithm of section 13.11 can be ap- 
plied to show that D\/^ is not a strange diagram whenever there exist disconnected Pfc's. More 
accurately, we have the following lemma. 

Lemma 11. Let T be a GSYT obtained by applying the algorithm of section \S.l\ to P>\j il - Suppose 
that for some k, Pk is disconnected. Then D\/^ has an amenable filling different from the one 
given by the algorithm. 

Proof. Let Pk = P 1 U • • • U P l , I > 1, where P l, s are mutually disjoint paths, ordered such that for 
any i, P l lies to the northeast of P' l+l . Note that the P ll s belong to mutually disjoint connected 

(k) 

components of the diagram D\L- Since the filling of Pk is obtained by the algorithm, the last box 

of P l is filled by a k. We claim that if we change it to a k, the new filling is still amenable. To 
this end we use Lemma El One can see that changing a k to a k does not affect fc-amenability of 
w(T)( k > unless we are changing the last box of P l . Moreover w(T)( k+1 ^ remains £;+ 1-amenable since 
w(T)( k+1 ^ can be written as u;(T)( fc+1 ) = w 1 ■ ■ ■ w l where vf is the row word of the intersection of 
Pk U-Pfc+i with the connected component of D^) which contains p* - *+l. A bookkeeping argument 

using the facts that the words w 1 , w 1-1 are k + 1-amenable and w is "not far from" being 
k + 1-amenable completes the proof. □ 

From Lemma ITTI we conclude that if Dx/^ is a strange diagram then for any k the set Pk which 
is obtained by the algorithm of section 13.11 is in fact a (connected) path. This fact simplifies the 
case-by-case analysis of possible shapes of D\//_ L - Probably a few remarks are necessary before 
starting the next section: 

•Amenability arguments. Throughout the next section, at several points we give procedures 
to modify an existing filling and claim that the new filling will be amenable too. The proofs of 
amenability of these new fillings are very similar in nature. However, most of these amenability 
proofs are not given because they are tedious but very simple. The main ideas are using Lemma 
Eland bookkeeping arguments. 

•Figures. Throughout the next section there are several illustrations which help the reader un- 
derstand the effect of procedures on the fillings. The figures usually demonstrate a union of 
Pm, Pm-i, •••) Pr f° r some r. Only the boxes whose entries change by the procedures are shown. 
The dots show that there may or may not be boxes in the direction of the dots. See the following 
example. 

Example 5. Let m denote the largest integer such that P m 7^ 0. Suppose m = 7 and the union 
Pq U Pi is illustrated by the diagram 



6 


6 


6 


7 



_6_ 

[6] 

This means that there may or may not be a vertical path of boxes which belong to Pq in the 
location of the dots. For instance Pq U P7 may actually be 
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6 


6 


6 


7 


6 

6 





or 



6 



Now again suppose m = 7 and the union Pq U P-j is represented by 



6 



6 



7 



This means that if the path Pq extends from the top right corner of Pq U P?, then its direction will 
be vertical in the beginning. (However, its direction does not have to remain vertical all the time.) 
For instance, Pq U P-j may actually be 
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6 


7 



or 
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7 



or 



Again let m = 7 and suppose Pq U Pi is illustrated by the diagram 
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but not 
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7 
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6 


6 


7 


6 





This means that Pq may extend either horizontally or vertically (but clearly not both, since Pq is 
a path). 

3.3. A case-by-case examination. Let T be the filling of D\/^ obtained by the algorithm of 
section I3~T1 By Lemma lTTl Pj. is a connected path for any k. Let m be the largest integer for which 
P m is nonempty. Then the following possibilities exist for P m : 

• P m consists of a single box. 

• P m lies within a single row and has at least two boxes. 

• P m lies within a single column and has at least two boxes. 

• P m is none of the above. 

We will study the first three possibilities in subsequent sections. Here we show that the fourth 
case is actually impossible when D\/^ is a strange diagram. 

Lemma 12. Suppose D\/^ is a strange diagram. Then the boxes of P m lie within a single row or 
column. 

Proof. The following figures show how to obtain a new amenable filling of D\/fj, when P m has a 
"turning point". If the path P m changes its direction at some point clockwise, i.e. if for integers 
x,y we have M x>y , M XtV+ i, M x _i ty £ P m then all we need to do is 

m + 1 and 



"x,y 



»x,y+l 



m. 



If it changes its direction counterclockwise, i.e. if 



the smallest y' such that ] 
Example 6. Suppose m 



»x,y,>®x,y-l,> 



x+i, y £ P m , then we should find 



»x,y 



i £ P m and then do 



x -,v' 



m + 1 and 



>>x,y'+l 



m. 



5. Then P m and the way it changes are illustrated below. 





5 






5 


5 


5 


. . . 


5 


6 



clockwise turn 
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5 5 



5 5 
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5 




5 


5 


5 




•w . . . 


6 





or 



Counterclockwise turn 

Checking amenability is easy and left to the reader. 

In the subsequent sections we address the other possibilities for P„ 



□ 



3.4. Case I: P m consists of one box only. We will assume Dx/^ is a strange diagram and 
m > 1. Let M x ^ y be the box in P m . Then by Lemma El and Lemma Q B^y+i contains m — 1 and 
B^-i^+i G Pm— l- Now we have the following two possibilities: 

1. M x ^i^ y ^ Then it follows that B^-i^+i should be the last box of P m _i, as P m -i 

cannot proceed to B x —2y+i by Lemma 121 This in turn yields three new possibilities: 
Case a. M x y +2, M x+ i <y+ i £ P m -\. In this case P m -i has two boxes only. 
Case b. B^+2 G P m -i- 

Case c. M X+1:V+1 G P m -i- 

should be filled by m — 1 and 



"rr— 1,3/ 



G I? 



Then 



-1,1/ 



G P 



m— 1 • 



One of the 



following six cases may take place. 



Case d. 
Case e. 

Case f. 
Case g. 
Case h. 

Case i. 



y>x-2,y £ Pn-1- 
^x-l,y-l,^x-l,y-2 € Pn-1 
"x— 1,J/— 1 

G P m _i but Baj-i.j, 

^-2,j/, 



i G P m _i but B x ._i j j / _ 2 £ D A/At . 

t Pn-i but Bjj+i^+i G P m -\. 

B>iE-l,2/-l ^ Pn-1 but B XjJ , + 2 G Pn-1- 
K a; -l, 2 /-l,B a ; ) j / +2,Ba;+l,2/4-l f Pn-1- 



Assuming m = 7 we can illustrate P m U P m -i i n cases a to i by the following figures. 
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Note that in the cases d,e,f,g and h given above we can modify the filling to get another amenable 
one as shown in the following figures. Therefore if P m U P m -i is one of those cases then D 
cannot be strange. . : 
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The only remaining possibility is Case i. If m = 2 then Dx/^ is equal to the underlying diagram 
of figure i, and has a unique amenable filling. Assume m > 2. Next we prove Proposition 1141 

Definition 13. A path of boxes is called a (p, q)-hook if it has p vertical boxes in the first column 
and q horizontal boxes in the first row. 

Example 7. The following figure illustrates a (2, 3)-hook. 



Proposition 14. Suppose Dx/^ is a strange diagram and P m —i U P m is as in Case i. Then for 
any j, Pj is a (pj,m — j + \)-hook for some pj < m — j + 1. Moreover, if for some j > 1 we have 
Pj < m — j + 1, then pj = Pj-i = ■ • • = p\. 

Proof. We use backwards induction. Assume the statement holds for 

Pm—r+l j •••) Pm 

where r > 1. We prove it for P m -r- Suppose that P m - r+i is an (/,r)-hook for some 1 < I < r. As 
before, assume B^y € Pm- Lemma II 11 Lemma ITI and Lemma 12*1 imply that 



1, we have 



»x—r,y+r—j 



eP n 



Obviously 
= Q. 



o For any j such that 1 < j < I 
and 

o For any j such that < j < r, we have M x _j^ y+r € P m -r- 

Let Q denote the path formed by the boxes in the two statements given above. 
Q ^ Pm-r- We have the following two cases: 

Case A. I < r. Then 7S> x -r,y+r+i-i should be the last box of P m -r- We will show that P„ 

Suppose, on the contrary, that P m -r ^ Q- Then either M x ^ y+r+ i £ P m ~r or B^+i^+r G 
P m -r- However, Dx/^ cannot be strange in either of the cases, as shown below. 
Case Al. M x+ \ w+T £ P m -r- Let u = x — r, v = y + r + l — I. Then the following procedure 
provides a new filling. 
Step 1 . Set t = 1 and M u v +\ <— m — r. 
Step 2. If B u+t -i, v -t+i $. D X /^ then stop. 

€ Dx/u- If there is no such y', then 



Step 3. Find smallest y' > v — t such that B n+t y 
stop. 

Step 4. If B u+i y £ P s then do M u+t -i )V -t+i <— s and 
Step 5. Increase t by one. Go back to Step 2. 

The idea behind the procedure can be more concretely described as follows. We elimi- 



>>u+t,v— 1+1 



s. 



nate the last box of P n 



add this box to D^™.. r+1 \ and find new paths P m _ r+ i, P n 



according to the algorithm of section 13.11 
The following figure demonstrates the above procedure when m 



8 and r = 3. 
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Case A2. B^+r+i £ P m -r- Then the following procedure gives the new amenable filling. 
Step 1 . Set t = r + 1. 

Step 2 . If t > 1 then replace the entry of B^ by that of B^ y+t— i. If £ = 1 then replace 

the entry of B^^+t by m. If t < 1 then stop. 
Step 3 . Decrease i by one. Go back to Step 2. 

The amenability of the diagram obtained by the procedure follows from statements 
similar to those that appear in the proof of Lemma ITU1 

For m = 7 and r = 2 the following figure demonstrates the effect of the procedure. 
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Case B. I = r. In this case if 
that P, 
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^ P m -r then 


we 


can 



in-r — Q- The following figure illustrates what happens when m = 7, r = 2 and 

G Pm—r- 
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Therefore we may assume ^> x —r : y G Pm—r- Next we show that 

Pm-r = QU {B^-r^}. 

Suppose on the contrary that P m -r 7^ QUlB^^y}. Then one of the following cases occurs. 
In all of them we show that D\/^ is not strange. 
Case Bl. B^— r— G P m -r- We can get a new amenable filling by the operations M x - r<y <— 

m — r + 1 and B, r _ rj j /+ i <— m — r. 
Case B2. M x - r ^ y -i G P m -r- In this case we can obtain a new amenable filling as follows. Find 
the smallest y' such that M x _ r ^ y i G P m -r- Then do ~B> x _ r ^ y / + i <— m — r, M x _ r ^ y / <— m 
and M x>y <— m. 

Case B3. M x ^ r ^\ ty ^ P m -r, ^>x—ry-i ^ Pm-r but B^+x^+r G P m -r- The new amenable filling is 
obtained by 

Bx-r,y <— m — r + 1 and B x _ ri3/+ i <— m — r. 
The following figure illustrates this procedure when m = 7, r = 4. 
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Case B4. B;c_ r _x l2/ ^ P m -r, ^>x-r,y-i ^ P m -r but B^+r+x G P m -r- In this case exactly the 
same procedure that was given in the analysis of Case A2 can provide a new filling. 
We illustrate it below with m = 7 and r = 4. 
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□ 



Remark. The proof of Proposition 1141 actually proves more. It proves the following corollary. 



Corollary 15. Let D\/^ be a strange diagram and P m U P m -i be as in Case i. Then X/fi = X//J, 
such that 

• X = {p + q + r>p + q + r — 1 > p + q + r — 2> ••• > p] and 11 
where p, q, r are integers such that p, q > 1, r > 0. 

• \ = {p + q>p + q— 1 > ■ ■ ■ > p + q — r} and ~fi = {q > q — 1 > • 
are integers such that p > and q > r > 0. 



{<? > q - 1 > • • • > 1} 

• > q — r} where p, q, r 
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Remark. The two cases of Corollary El may overlap. Moreover, not all of them are such that 
P m U Pm-i is as in Case i. However, we have written it in the given form above so that we can 
refer to it at other points of this manuscript. For example see Corollary 1201 

Next we analyze cases a,b and c which were introduced at the beginning of section 13.41 
Proposition 16. If D\/^ is a strange diagram and P m U P m -i is as in Case b then we have 

X = {m > m — 1 > • • • > 1} and \i = {fix > • • • > fn} where I < m — 1. 



Proof. It suffices to show that there does not exist any z such that M x +i jZ G -^a/a* 
contrary, and let z be the smallest integer such that M x+ i tZ G Let r be such tha 

From Lemma|21 Lemma[B]and Lemma ITT1 it follows that M x z G P r . Moreover, by LemmaEJit follows 
that for any s such that r < s < m, 



Assume the 
^>x+l,z £ Pr- 



B;c— m+s+l,2/+m— s+1) ™a;— m-ys-y\,y+m— si ™a!— m+s,j/+m— s € ■*«■ 

Taking s = r implies that z > y + m — r and the path P r makes a clockwise turn at some point. 

Let u, v be such that M U)V , B M __i jlM B Ui „ + i G P r and v < z. For example we can take u = 
x — m + r + l,v = y + m — r. It follows from Lemma|7|that for any t > such that t + r < m, 

B u +M-*> ^u+t-l,v-t, ^u+t,v-t+l G Pr+t- 

Now the following procedure provides a new amenable filling. 
Step 1 . Set t = 0. 

Step 2 . If u + t < x then do B n+ti „-t ^ r + t + 1 and M u+t +i )V -t <— r + t + l. If u + 1 = x and the 
entry of M u+tjV -t-i is either w or w where w £ A, then do M u+t:V -t <—w. If it + 1 > x then 
stop. 

Step 3 . Increase t by one. Go back to Step 2. 
The idea of the procedure is more concretely explained as follows. We eliminate one of the boxes 
of P r , add it to D\, , and then find new paths P r +i, Pm according to the algorithm of section 



rm 



The following figure demonstrates how the procedure can be applied to a case with m = 7 and 
r = 4. In fact it can be seen that there are at least two ways to get new amenable fillings. 
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□ 



Proposition 17. Xei D\/^ be a strange diagram. Assume P m U P m ~\ is as in Case a or Case c. 
Then D^/^ is given by one of the following cases. 



'A/A« 



is given by partitions of Proposition MfA 



y \/fi = D\ f or an arbitrary X. 

Proof. Let, as before, M XtV G P m . Suppose D\/^ is not given by partitions of Proposition [TBI 
According to the proof of Proposition^!! this means that there exists a t such that M x+ \ tt G D\/^- 
We can assume t is the smallest integer with this property. Let r be such that B^+i^ G P r . Let r' 
be the largest integer for which P r > does not lie within a single row. Note that if there is no such 
r' then it follows that Dx/^ = Dx- 

Set r\ = min{r, r'}. An argument similar to that of Proposition 1161 shows that P ri should make 
a clockwise turn; i.e. there exist p, q such that M P g, B p _ig, B P9 +i G P ri . Now one of the following 
possibilities happens. 

Case CI. r > r\. Then it follows that r > r' . In this case we use the following procedure. 
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Step 1 . Set t = 0. 

Step 2 . If B Pi g +t G Dx/fj, then do B Pi(?+t+ i <— B P)9+t . Otherwise, do B Pj(?+t+ i <— ri — i and stop. 
Step 3 . Decrease i by one. Go back to Step 2. 

The idea of the procedure is similar to that of the procedure given in Case B4 of Proposition 

M 

Case C2. r = r%. Then it follows that r < r'. In this case we use the following procedure. 

Step 1 . Set t = 1 and do B Pi(? +i <— T\. 

Step 2. If Bp +tjg _ t G Dx/fj,, then do B p+i _i >g _ i+ i <- ri + i and B p+ti9 _ t+ i <- n + i. 

Step 3. If B p+ t >g _ t ^ Dx/fj, and B p+ t_i )(? _t G jD A /^ then do B p+t _i ]? _ m <- r x + 1 and stop. 

Step 4. If B p+ti9 _ t , B p+t _i jg _ t ^ then do B p+t _i j9 _ t+ i <- r 1 + i and stop. 

Step 5. Increase t by one. Go back to Step 2. 

The idea behind the procedure is to eliminate one of the boxes of P ri , modify the filling of 
P n slightly, attach the box removed from P ri to D^/^ and find new paths P ri +i, ■■■,P m 
according to the algorithm of section 13.11 

□ 

3.5. Case II: P m lies within a single row and has at least two boxes. Let us assume that 
the boxes that belong to P m are M x -i tV , ...,M X y where I > 0. From Lemma[H]it follows that for any 
j G {0, I + 1}, B x _j >y+ i € P m -l- We have the following two cases. 

Case Dl. M x -i^i^ € Dx/p- In this case the boxes Bj.—j—i^+i, ...,M X) y+\ and M x -i-.x t y form a 
path in P m -\. An argument similar to the one given in Case B of Proposition 1141 proves that there 
are no other boxes in P m -\- Next we state the following proposition which is similar in statement 
and proof to Proposition 1141 

Proposition 18. Suppose that D\/n is a strange diagram such that P m U P m -i is as in Case Dl. 
Then for any j G {1, m}, Pj is a (j)j,l + m — j + l)-hook for some pj < m — j + 1. Moreover, if 
there exists a j > 1 such that pj < m — j + 1, then pj = Pj-i = ■ ■ ■ = p\. 

We omit the proof of Proposition 1181 because it is very similar to the proof of Proposition 1141 

Case D2. M x -i-i tV ^ D\/n- The analysis of this case is similar to that of Case a in section l3~4l 
(See Proposition 1171) In fact we can prove the following proposition. 

Proposition 19. Suppose that Dx/^ is a strange diagram which is as in Case II above, and 
moreover M x _i^i tV ^ Dx/fj,- Then Dxu = Dj for an arbitrary A. 

Proof. It suffices to show that for any j, Pj lies within a single row. The proof is by contradiction. 
Assume the contrary, and let j be the largest integer such that Pj does not lie within one row. From 
Lemma |H1 it follows that if p 1 is the largest integer for which there exists a box B p / >2/+m _j_i G Pj+i 
then there exists a p > p' such that B Pj? , B p _i ](? , B Pig+ i G Pj, where q = y + m — j. Now set r\ = j 
and apply the procedure given in Case CI of the proof of Proposition El to get a new amenable 
filling. □ 
Corollary 20. Let Dx/^ be as in Case II. Suppose Dx/^ is a strange diagram. Then either 

Dx/fj, = Dj for an arbitrary A, or Dxip, is given as in the statement of Corollary \15i 

3.6. Case III: P m lies within a single column and has at least two boxes. The analysis in 
this case is pretty similar to the previous cases. Suppose P m consists of the boxes M x>y+ i, ...,M XtV 
where I > 0. Then for any j G {1, + 1}, M x _i jV+ j G P m -\- Moreover, M XjV+ i + i G Pm-l- Next 
we show that none of the boxes B^+i^+i+i, B x _i j,_i, M x _2,y can belong to P m -i- In fact in each 
of the following cases we show that Dx/^ cannot be a strange diagram. 

• M x -i t y-i G Pm-i- Then we get a new filling as follows. Find the smallest y' such that 
B^-iy G P m -i and do 

Bx-iu'+i <— m — l,B x _i y/ <— m and M x>y <— m. 
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• M x _2,y S Pm-i- Then we obtain a new filling by 

^>x— 1 y *~ m an d ^>x-l,y+l *— m — 1. 

• ^>x-i,y-i> ^>x-2,y ^ Pm-i but M x+ i >y+ i + i G P m -i- Then a new filling can be obtained as in 
Case Al or Case B3 of Proposition 1141 We leave the details to the reader. 

Proposition 21. Let D^/^ be a strange diagram. Suppose B^-i^ ^ P m -\- Then 

A = {m > m — 1 > • • • > 1} and = {fii > - • • > fii} 

where I < m — 1 . 

Proof. It suffices to show that there does not exist any y' such that B x+ iy £ Suppose there 

exists such a y' , and without loss of generality assume that y' is the smallest integer with this 
property. Suppose B^y £ P r . Then by Lemmas and |H1 it follows that 

B^— m+r,y+m— r > B^— m+r,y+m— r+1 £ -Pr- 

Now the following procedure provides a new amenable filling. 
Step 1 . Set j = r and do M x _ m+rt y +m _ r+1 <- r. 

Step 2. If j < m then do Ba;_ m+: / ia+m _j <- j + 1 and Bj;_ m+;? - +l!a+m _ 3 - <— j + 1. 
Step 3 . If j = m then do B^m+j^+m-j <— m + 1 and stop. 

Step 4. Increase j by one. Go back to Step 2. ^ 

The proof of the following proposition is very similar to those of Propositions and 1141 and 
therefore we omit its proof. 

Proposition 22. Let Dx/^ be a strange diagram given as in Case III and suppose B x _i iS/ 6 P m -i- 
Then A,/i are given as in Corollary \1 51 

Corollary 23. Let D\/fj, be a strange diagram given as in Case III. Then D\/^ is given either by 
the partitions of Corollaru \15\ or by the partitions of Provosition \21\ 

3.7. Proof of uniqueness. In the previous sections we showed that the only diagrams which 
could possibly be strange are those listed in Theorem HO In this section we prove that all of those 
diagrams are indeed strange. First we need a simple property of any arbitrary GSYT. 

Lemma 24. Let T be an amenable GSYT and let w(T), the row word ofT, have length n. Then 
for any k > 1 such that mk-\{n) > we have rrik(n) < rrik-i{n). 

Proof. From Definition |3] it follows that for any k > 1, nik(n) < nik-i(n). Suppose mk-\{n) = 
mfc(n). Then it follows from Definition |3] that if jk (respectively jk-i) is the smallest integer such 
that Wj k = k (respectively Wj k _ 1 = k — 1) then jk < jk-i- The fourth part of Definition implies 
that Wj^k — 1 for any j < jk-i- Since mk-i(j) > rrik(j) for any j G {0, 2n — 1}, it follows that 
m k(n + jk-i — 1) = mk-i( n + jk-i ~ 1) = m k-i( n )- But then Wj k _ 1 = k — 1, which contradicts the 
second part of Definition |31 □ 
It is obvious that if D\/^ = D\ then Dx/^ is strange. Next we prove that if D\/^ is given by 
partitions of Corollary 1151 then D^/^ is a strange diagram. We will present the argument only for 
the case \ = {p + q + r>p+q + r — 1 > p + q + r — 2> ••• > p} and fj, = {q > q — 1 > • • • > 1} 
where p, q, r are integers such that p, q > 1, r > 0. The other case can be treated in a very similar 
fashion. 

Proposition 25. Let Dxjp, be given by\ = {p + q + r>p + q + r — 1 > p + q + r — 2 > ••• > p} 

and H = {q > q — 1 > • • • > 1} where p, q, r are integers such that p, q > 1, r > 0. Then Dx/^ is a 
strange diagram. 

We give the proof of Proposition |2H1 through Lemma I2fi| Lemma EH and Lemma 1281 Let us 
assume that M x>y is the top right box of the diagram L>x/n- Let n denote the number of boxes 
of D\/^- We consider an arbitrary amenable filling of Dx/^ and show that it has to be the one 
obtained by the algorithm of section l3~Tl 
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Lemma 26. For any integers k,x' such that k>0 and x—p — r + 2 + k <x'<x, if^ x ',y-k G D\/^ 
then it should be filled by a k + 1. 

Proof. We prove this lemma by induction on k. For k = 0, the above statement follows from the fact 
that we should have ~B x y G {1,1} and moreover no two l's can lie within the same row. Next assume 
k > 1. Let x' be chosen as above. Then by induction hypothesis, the boxes M x ^ y _k+i,^ x '-i,y-k+i 
are filled by k. Moreover, none of the rows which lie above the box M X y—k can contain an element 
of A which is strictly larger than k, because their rightmost boxes are filled by elements less than 
or equal to k. Consequently, B^-jfe should be filled by an element of {k + 1, k + 1}. (Note that 
M x y-k+l contains a k.) This in turn limits the entries of ^> x ' y-k and ^x'—iy—k to {k + 1, k + 1}. 
However, if "M>x',y-k is filled by a k + 1 then M x i_i yy _k should also be filled by a k + 1, which is a 
contradiction by Definition ^ Therefore M x ^ y -k has to be filled by a k + 1. □ 

Lemma QUI implies that for any k > 1, if there is at least one box in -Da/^ filled by a k, then 
there are at least p + r — k boxes in D^/^ which are filled by a A;. Note that since D^/^ has p + r 
columns, there can be at most p + r boxes which are filled by a 1. Lemma l2H implies that there 
can be at most p + r — k + 1 boxes which are filled by a k. We will see below that actually the 
latter possibility takes place. More accurately, we show that if there is at least one k in the filling 
of D\/im, then there are exactly p + r — k + 1 of them. 

Lemma 27. The largest integer that appears in the filling of Dx/ P is min{p + r, q + r + 1}. For 
any k G {1, ...,mm{p + r,q + r + 1}} there are exactly p + r — k + 1 boxes in D\/^ which contain a 
k. 

Proof. If p + r > q + r + 1 then by Lemma[2f)]the box M Xiy _ q _ r is filled by a q + r + 1. But B Ifl _ g _ r is 
the rightmost box of the lowest row of Dx/ P , and since the rows and columns of a filling are weakly 
increasing, all of the boxes have to be filled by elements of A which are less than or equal to q+r + 1. 
Now Lemma implies that M x+q ^ p+ 2,y-q-r contains a q + r + 1 and ^ x + q - p +i.y- q - r +i contains a 
q+r. Because of the shape of Dwj, we can see that M x+q - p+ i ty - q - r G Since M x+q - p+ i ty - q - r+ i 

contains a q+r, the entry of M x+q - p+ i jy - q -. r should belong to {q + r + 1, q+r+1}. But the entry of 
Mx+g^p+i^y-q^r cannot be q + r + 1 because this implies that the leftmost occurence of an element 
of {q + r + 1, q + r + 1} in the row word of the filling is marked. Consequently, M x+q ^ p+ i ty ^ q ^ r is 
filled by a q + r + 1 which implies that m g + r +i(n) > p — q. Now we have 

p + r > m\{n) > • • • > m q+r+ i(n) > p — q 

which implies that mfc(n) = p + r — k + 1 for any k G {1, q + r + 1}. 

If p + r < q + r+1, then Lemma implies that B x . jJ/ _ p _ r+ 2 has to be filled by p + r — 1. 
Now M X)y _ p - r+ i G Dx/fM and therefore it should be filled by an element of A which is larger than 
p + r — 1. Consequently, m p+r (n) > 0. Now by Lemma I2H we have 

p + r > mi(n) > • • • > m p+r (n) > 1 

which implies that mfc(n) = p + r + 1 — k. In particular, mfc(n) > if and only if k < p + r. □ 

Let M = mm{q + r + 1, p + r}. By Lemma 1771 for any 1 < k < M there are exactly p + r — k + 1 
boxes which contain a k. Lemma 123 determines the position of p + r — k of these boxes uniquely. 
Therefore for any k < M, the location of precisely one box containing a A; is left to be determined. 
In fact it turns out that after the location of this box is determined, the entire filling is also 
determined uniquely. See Lemma l28l below. 

Lemma 28. Let x' G {x,x — 1, ...,x — p — r + 1} and let y' be the smallest integer such that 
^>x',y' G Dx/n- Then 

• If x' < x — p — r + M then ivy has to be filled with an x' — x + p + r . 

• For any y" such that y' + 1 < y" < y — x' + x — p — r + 1, the box M x ^ y f has to be filled with a 
x' — x + p + r. 
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Proof. Let b(x') be the entry of the box M x > t yi of the statement of the lemma. Since the rows and 
columns are weakly increasing we have 

b(x — p — r + 1) < b(x — p — r + 2) < • • • b{x — p — r + M) 

where the inequalities are interpreted in the ordering of A. From Definition [3] it follows that 
b{x — p — r + j) should be unmarked for any j £ {1, M}. In fact assume on the contrary that 
b(x — p — r + j) =1 for some I. Let x' = x — p — r + j and y' be such that b(x — p — r + j) be 
the entry of the box B^y. By Definition [3] there should exist a box B^/y/ such that y" < y' and 
~$>x",y" is filled by an From the shape of D\/^ it follows that x" > x'. But then B^/y € D\/^ 
and it is impossible to fill the latter box such that Definition []"] holds. 

Since b(x — p — r + j) is unmarked for any j £ {1, M} and there is exactly one box filled by 
a j whose location is not determined by Lemma I2ffl it turns out that b(x — p — r + j) = j for any 
such j. 

Finally, M x > y " lies below a box filled by an x' — x + p + r — 1 and above a box filled by an 
x' — x + p + r. Consequently, its entry has to be x' — x + p + r. 

□ 

The proof of Proposition is completed by Lemma 1281 The following proposition completes 
the proof of Theorem [HJ 

Proposition 29. Let D^/^ be given by X = {m > • • • > 1} and fi = {fi\ > • • • > /^} such that 
I < m — 1. Then D\/^ is a strange diagram. 

The rest of this section is devoted to the proof of Proposition EHl Suppose Daa* nas P columns 

and n boxes. This means that p = m — I and n = m ^™ +1 ^ — (/ii + • • • + Hi). Let w = w(T) be the 
row word of the filling of D^/^- w can be expressed as w = w\ ■ ■ ■ w n where each uii € A. Since 
each column of Dx/^ contains at most one box filled by 1, m\{n) < p. Lemma |^] implies that 

(3) mk{n) < p — k + 1 for any k E {1, ...,p}. 

Let M X)V denote the box in the lowest row of Dx/^- (Note that there is only one box in the lowest 
row.) For any i € {0, ...,p — 1} let bi denote the entry of the box B^-j^+j. Since rows and columns 
are weakly increasing, the b^s form a decreasing sequence in A; i.e. 

(4) &o > • ■ ■ > ifc-l- 
Lemma 30. 6j = p — i for any i € {0, ...,p — 1}. 

Proof. First we show that all the b^s are unmarked. Suppose on the contrary that for integers i 
and I, bi = I. Then by Definition |31 there should be a box M UfV such that u > x — i, v < y — i and 
the entry of M u>v is I. But then it will be impossible to choose an entry for B Uj3/ _j so that Definition 
^ holds. Therefore 6j cannot be marked for any i. From equation © it is clear that &o < P- 

It is now easy to use (J3J to prove that bi = p — i. □ 

Definition 31. A diagonal D s in Dx/^ is the set of boxes given by 

D s = {B a ./y \x + y' = x + y + s} n D x/tl . 

If D s ^ then we can express it as 

D s = {B Xsi y s ,M Xa+lj y s -l,...,B X3+ l 3> y s -l s } 

for some integer l s > 0. 

Our approach is to prove inductively that the entries of boxes in every D s are uniquely deter- 
mined. The basis of induction is s = 0, which follows from Lemma [3*01 

Lemma 32. Fix i S {0, ...,p — 1}. Let z > y + i be an integer such that M x _i tZ E Dx/^- Then the 
entry o/B x _j i2 is an element of A strictly less than p — i. 
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Proof. This is because B 2 _i 2 lies above M x _i tV+ i which is filled by a p — i. □ 
Now let D s and D s+ \ be two successive nonempty diagonals of D\/^. We can express them as 
D s = {^x a ,y a ,—,^x B +h,y a -l a } an d D s+1 = {M Xg+uyg+1 , ^>x s+1 +l s+1 ,y a+1 -l s+1 } 
where l s , l s +i > 0. It can be seen that one of the following cases can happen: 

• x s+ i = x s + 1 and y s +\ = Us- In this case l s+ i = l s — 1. 

• x s+ i = x s and y s +\ = y s + 1- In this case l s+ \ = l s . 

From Lemma """01 it follows that 6o = p, i.e. m p (n)=l. Now Lemma OH implies that mk(n) = 
p — k + 1. The following lemma now follows immediately from Lemma l"""21 

Lemma 33. For any k such that p > k > 1 and any u > x — p + k there exists an integer v such 
that M uv is filled by a k. 

Next we prove the following lemma, which completes the proof of Proposition 1291 

Lemma 34. The entries of the boxes of D s+ \ are given according to the following rules: 

• If y s+ i=y s + l then for any i £ {0, l s +l} the entry of ^x s+ i+i,y 3+ i-i isi + 1. 

• If x s+ i = x s + 1 then for any i G {0, ...,l s +l} the entry of "®>x a+x +i,Vs+i-i is i + 1. 

Proof. We use induction on i. Namely, we assume that the entries of the boxes in D s are given 
according to the rules stated in the lemma, and then we use induction on i to prove that the entry 
of ^> Xa+ i+i,y s+1 -i is given according to the rules in the statement of the lemma as well. 

Case 1. Obviously 13>x s+1 ,y a+1 = 1, since it lies above the box B Xsi3/s whose entry belongs to {1, 1}. 
This amounts for the basis of induction, i.e. i = 0. Next suppose i > 0. Then B a , s+1+ j j j /s+1 _j lies to 
the right of B Xs+ j_i >ys _j + i and above M Xs+ i tVa ^i. However, by our assumption about D s , the entries 
of the latter two boxes, which belong to D s , are known. A simple argument based on monotonicity 
of rows and columns implies that if b is the entry of B a ; ij+1 _|_j j?/a+1 _j then b € {i,i + 1}. It suffices 
to show that b = i is impossible. The proof is by contradiction. Suppose on the contrary that 
b = i. By induction hypothesis we know that B Xs+1 _|_j_i j j /s+1 _,; + i is filled by i, which implies that 
it is impossible to choose an entry for M Xg+1+ i t y g+1 _i + i such that Definition ^ holds. Consequently, 
b = TTT. 

Case 2. First note that 9 Xa+1 +i,y a+ i-i lies to the right of M Xg+ i jys _i and above B Xs+ i + i jJ/s _i_i. 
But the entries of the latter two boxes are known, and a simple argument based on monotonicity 
of rows and columns implies that if 6 is the entry of B Xs+1+ j iys+1 _j then b S {i + l,i + 2}. Next 
we show that b cannot be equal to i + 2. Suppose on the contrary that b = i + 2. Suppose b 
corresponds to the letter w ui of the row word of the filling. Then by a bookkeeping argument we 
have 

m i+ i(n + tii) = rrii+2(n + Ui). 

By Lemma 152*1 and Lemma 1551 there exists a z such that the box M Xg+1+ i jZ is filled by an i + 1. 
Obviously z > y s +i — i- Let the entry of M x s+1 +i, 2 correspond to w u . Then there should exist an 
integer v! such that u\ < u' < u and w u i = i + 1, because otherwise we will have mj+i(n + u — 1) = 
m i+2( n + u — 1) and w u = i + 1 which contradicts the second property of Definition [31 

However, existence of v! leads to a contradiction too. In fact w u > should correspond to a box 
Bj.» y/i such that either y" < z or y" = z and x" < x s+ i + i. However, if x" < + i and y" > 
y s +i — i then Bj-w „// lies above M Xg+1+ j t y g+1 _j for some j < i, and therefore the entry of B^/y/ should 
be strictly less than i + 1. (Note that if i > then by induction hypothesis B Xs+1+ j_i i j /s+1 _j + i = i 
and if i = then B Xs+1+ j_i i j /s+1 _j + i ^ D\u-) If ^" = ^s+i + i and y" < z then B^/y/ lies below 
B Xs+ i+j,2 and therefore its entry should be strictly larger than i + Finally, if x" > x s+ i + i and 
y" < z, then B^//^ E ^A//^ it is impossible to choose its entry such that Definition [*Q holds. 
Therefore all of these cases lead to a contradiction. 

□ 
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